Unsupervised Discovery of Facial Events: Learning a Dynamic Vocabulary for Facial Analysis

Automatic facial image analysis is a long standing research problem in computer vision. A key component in facial image analysis, largely conditioning the success of subsequent algorithms (e.g. facial expression recognition), is to define a vocabulary of possible dynamic facial events. To date, that vocabulary has come from the anatomically based Facial Action Coding System (FACS) or taxonomies derivative of FACS (e.g., EM-FACS and basic emotions). Each FACS action unit (AU) corresponds to the movements of one or more facial muscles. The aim of this paper is to discover a taxonomy of facial movements directly from video, without recourse to FACS or its derivative. Using unsupervised learning, we discover dynamic facial events directly from video of naturally occurring facial behavior (i.e., not posed) of multiple people. Several issues contribute to the challenge of this task. These include non-frontal pose, moderate to large out-of-plane head motion, large variability in the temporal scale of facial gestures, person variability and the exponential nature of possible facial action combinations. To address these problems, this paper proposes a novel temporal segmentation and multi-subject correspondence algorithm for matching expressions. Our method achieved good convergent validity with manual FACS annotation. To the best of our knowledge, this is the first attempt to learn a dynamic vocabulary of facial events from video.